Automatic Discovery of Popular Landmarks

ABSTRACT

In one embodiment the present invention is a method for populating and updating a database of images of landmarks including geo-clustering geo-tagged images according to geographic proximity to generate one or more geo-clusters, and visual-clustering the one or more geo-clusters according to image similarity to generate one or more visual clusters. In another embodiment, the present invention is a system for identifying landmarks from digital images, including the following components: a database of geo-tagged images; a landmark database; a geo-clustering module; and a visual clustering module. In other embodiments the present invention may be a method of enhancing user queries to retrieve images of landmarks, or a method of automatically tagging a new digital image with text labels.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of, and claims priority to, U.S. patentapplication Ser. No. 12/119,359, filed May 12, 2008, the entire contentof which is incorporated herein by reference.

BACKGROUND

This invention relates in general to digital image collections, and moreparticularly, to identifying popular landmarks in large digital imagecollections.

With the increased use of digital images, increased capacity andavailability of digital storage media, and the interconnectivity offeredby digital transmission media such as the Internet, ever larger corporaof digital images are accessible to an increasing number of people.Persons having a range of interests from various locations spreadthroughout the world take photographs of various subjects and can makethose photographs available, for instance, on the Internet. For example,digital photographs of various landmarks and tourist sites from acrossthe world may be taken by persons with different levels of skill intaking photographs and posted on the web. The photographs may show thesame landmark from different perspectives, and taken from the same ordifferent distances.

To leverage the information contained in these large corpora of digitalimages, it is necessary that the corpora be organized. For example, atdigital image web sites such as Google Photos or Picasa, starting at ahigh level menu, one may drill down to a detailed listing of subjectsfor which photographs are available. Alternatively, one may be able tosearch one or more sites that have digital photographs. Some touristinformation websites, for example, have downloaded images of landmarksassociated with published lists of popular tourist sites.

However, there is no known system that can automatically extractinformation such as the most popular tourist destinations from theselarge collections. As numerous new photographs are added to thesedigital image collections, it may not be feasible for users to manuallylabel the photographs in a complete and consistent manner that willincrease the usefulness of those digital image collections. What isneeded therefore, are systems and methods that can automaticallyidentify and label popular landmarks in large digital image collections.

SUMMARY

In one embodiment the present invention is a method for populating andupdating a database of images of landmarks including geo-clusteringgee-tagged images according to geographic proximity to generate one ormore geo-clusters, and visual-clustering the one or more gee-clustersaccording to image similarity to generate one or more visual clusters.

In another embodiment, the present invention is a system for identifyinglandmarks from digital images, including the following components: adatabase of geo-tagged images; a landmark database; a geo-clusteringmodule in communication with said database of geo-tagged images, whereinthe geo-tagged images are grouped into one or more geo-clusters; and avisual clustering module in communication with said geo-clusteringmodule, wherein the one or more gee-clusters are grouped into one ormore visual clusters, and wherein visual cluster data is stored in thelandmark database.

In a further embodiment the present invention is a method of enhancinguser queries to retrieve images of landmarks, including the stages ofreceiving a user query; identifying one or more trigger words in theuser query; selecting one or more corresponding tags from a landmarkdatabase corresponding to the one or more trigger words; andsupplementing the user query with the one or more corresponding tags,generating a supplemented user query.

In yet another embodiment the present invention is a method ofautomatically tagging a new digital image, including the stages of:comparing the new digital image to images in a landmark image database,wherein the landmark image database comprises visual clusters of imagesof one or more landmarks; and tagging the new digital image with atleast one tag based on at least one of said visual clusters.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

Reference will be made to the embodiments of the invention, examples ofwhich may be illustrated in the accompanying figures. These figures areintended to be illustrative, not limiting. Although the invention isgenerally described in the context of these embodiments, it should beunderstood that it is not intended to limit the scope of the inventionto these particular embodiments.

FIG. 1 is a system to populate and update a landmark image databaseaccording to an embodiment of the present invention.

FIG. 2 shows a high level flowchart of a method implementing anembodiment of the present invention.

FIG. 3 is a flowchart showing more detailed operation of ageo-clustering stage shown in FIG. 2, in one embodiment.

FIG. 4 is a flowchart showing more detailed operation of a geo-clustercreation stage shown in FIG. 3, in one embodiment.

FIG. 5 is a flowchart showing more detailed operation of avisual-clustering stage shown in FIG. 2, in one embodiment.

FIG. 6 is a graphical user interface used in one embodiment of thepresent invention.

FIG. 7 is a method of updating a landmark image database according to anembodiment of the present invention.

FIG. 8 is a method of enhancing user queries using stored landmarkinformation, according to an embodiment of the present invention.

FIG. 9 is a method to automatically annotate images containinglandmarks, according to an embodiment of the present invention.

FIG. 10 is an example user interface screen, according to an embodimentof the present invention, showing information about landmarks andcorresponding clusters, retrieved according to user-specified selectioncriteria.

FIG. 11 is a flowchart illustrating the operation of a method tomaintain clusters and landmarks according to an embodiment of thepresent invention.

FIG. 12 is an example user interface screen showing details about onevisual cluster, according to an embodiment of the present invention.

FIG. 13 is a flowchart illustrating the operation of a method tomaintain visual clusters according to an embodiment of the presentinvention.

DETAILED DESCRIPTION OF EMBODIMENTS

While the present invention is described herein with reference toillustrative embodiments for particular applications, it should beunderstood that the invention is not limited thereto. Those skilled inthe art with access to the teachings herein will recognize additionalmodifications, applications, and embodiments within the scope thereofand additional fields in which the invention would be of significantutility.

The present invention includes methods and systems for automaticallyidentifying and classifying objects in digital images. For example,embodiments of the present invention may identify, classify andprioritize most popular tourist landmarks based on digital imagecollections that are accessible on the Internet. The method and systemsof the present invention can enable the efficient maintenance of anup-to-date list and collections of images for the most popular touristlocations, where the popularity of a tourist location can beapproximated by the number of images of that location posted on theInternet by users.

A popular landmark recognition system 100 according to an embodiment ofthe present invention is shown in FIG. 1. Processing module 101 includesa geo-clustering module 102 and a visual clustering module 103. Thevisual clustering module 103 may also include a popularity module 104.The processing functionality of modules 102-104 is described below: thegeo-clustering module 102 is described with respect to FIGS. 3-4; andthe visual clustering module is described with respect to FIG. 5. Theprocessing functionality of modules 102-104 may be achieved in software,hardware or a combination thereof. For example, modules 102-104 may beimplemented entirely as software modules, or some of the functionalityof the geo-clustering module 102 may be implemented using hardware suchas a field programmable gate array (FPGA). It will be understood by aperson of skill in the art that processing module 101 may includeadditional components and modules that facilitate the functions of thepresent invention. For example, processing module 101 may include one ormore processors, a memory, a storage device, modules for interfacing toexternal devices including the graphical user interface 130, thegeo-tagged image corpus 110, and the landmark database system 120.

The landmark database system 120 may include a landmark database 121 andassociated indexes 122. The landmark database system 120 may beco-located on the same processing platform as module 101 or may beseparately located. The landmark database 121 may include a collectionof landmarks recognized by the system 100. The information stored foreach landmark in landmark database 121 may include images or a list ofimages of the landmark, image and feature templates, and metadata fromthe images including geo-coordinates, time, and user information. Thelandmark database 121 may also contain the visual clustering andgeo-clustering data required for the processing in processing module101. The indexes 122 may include indexing that arranges the landmarks inlandmark database 121 in order of one or more of, for example andwithout limitation, popularity, geographic region, time, or other userdefined criteria as subject of interest. The link 141 may be any one ora combination of interconnection mechanisms including, for example andwithout limitation, Peripheral Component Interconnect (PCI) bus, IEEE1394 Firewire interface, Ethernet interface, or an IEEE 802.11interface.

A user interface 130 allows a user or other external entity to interactwith the processing system 101, the landmark database system 120, andthe geo-tagged image corpus 110. The user interface 130 may be connectedto other entities of the system 100 using any one or a combination ofinterconnection mechanisms including, for example and withoutlimitation, PCI bus, IEEE 1394 Firewire interface, Ethernet interface,or an IEEE 802.11 interface. One or more of a graphical user interface,a web interface, and application programming interface may be includedin user interface 130.

The geo-tagged image corpus 110 may include one or more digitalgeo-tagged image corpora distributed across one or more networks. Aperson skilled in the art will understand that the corpus 110 may alsobe implemented as a collection of links to accessible geo-tagged imagecollections that are distributed throughout a network. The corpus 110may also be implemented by making copies (for example, downloading andstoring in local storage) of all or some images available in distributedlocations. In some embodiments, a part of the geo-tagged image corpusmay exist on the same processing platform as the processing system 101and/or landmark database system 120. The different collections ofgeo-tagged images that constitute the geo-tagged image corpus 110 may beinterconnected through the Internet, an intra-network or other form ofinter-network. The processing system 101 takes as input, images madeavailable from the geo-tagged image corpus. In some embodiments, theimages from the distributed image collections may be converted to astandard graphic format such as GIF, either upon being stored in corpus110 or before being input to processing module 101. Embodiments may alsorequire that other forms of standardization, such as reduction orenhancement of resolution, or processing is performed on images prior toeither upon being stored in corpus 110 or before being input toprocessing module 101. The corpus 110 may be connected to othercomponents of the system by links 142 and 143 using any one or acombination of interconnection mechanisms including, for example andwithout limitation, PCI bus, IEEE 1394 Firewire interface, Ethernetinterface, or an IEEE 802.11 interface.

FIG. 2 is a flowchart of a process 200 of an embodiment of the presentinvention that creates or updates a database of landmarks 121 usinggeo-coded images from a image corpus 110. Process 200 includes twoprimary processing stages: a geo-clustering stage 201, and a visualclustering stage 202. Given a collection of geo-coded digital images,for example, a large collection of digital images of various touristdestinations, a geo-clustering stage 201 may divide the available imagesinto separate groups based on the geo-location codes of each photograph.The geo-clustering stage makes use of the geo-coding available in eachphotograph to make a relatively quick separation of the images todifferent groups or geo-clusters. Pre-configured parameters, including adefault radius within which images are considered to belong to the samegeo-cluster may be utilized. The geo-clusters generated in thegeo-clustering stage 201 are then input to the visual clustering stage202. In the visual clustering stage 202, the system attempts to separatethe images in each geo-cluster by subdividing into clusters of images ofthe same object or landmark (i.e., visual clusters) based on imagesimilarity. Note that in general, geo-clustering of a collection ofphotographs is computationally less expensive than visual clustering ofthe same collection of images, due at least in part to the former beinga comparison of geo-location information already included in eachphotograph. In contrast, for example, visual clustering 202 may includeperforming object recognition, feature vector generation and comparisonfor each identifiable object in each of the images, and then comparingthe feature vectors of different images.

In some embodiments, visual cluster information including the associatedimages and/or references to associated images may be stored in adatabase such as landmark database 121. The images and/or the virtualimages stored in landmark database 121 may be accessible using one ormore indexes 122 that allow access to stored visual clusters based onconfigurable criteria including popularity. For example, the storedvisual clusters may be processed by a popularity module 104 that updatesan index 122 to allow access in order of the number of unique users thathave submitted images to each cluster.

In some embodiments, selected visual clusters may be subjected to reviewby a user and/or may be further processed by a computer program. Forexample, optionally, visual clusters satisfying specified criteria, suchas, having less than a predetermined number of images, may be subjectedto review by a user. A user may modify one or more visual clusters byactions including, deleting an image, adding an image, or re-assigningan image to another cluster. A user may also specify new tag informationor modify existing tag information. A person skilled in the art willunderstand that processing the visual clusters according to externaldata received from a user or a computer program may require the systemto perform additional functions to maintain the consistency of thegeo-cluster and visual cluster information stored in the database system120.

FIG. 3 shows two processing stages, create geo-clusters 301 and validategeo-clusters 302, that are included in the geo-clustering stage 201 insome embodiments of the present invention. Creating geo-clusters 301 mayinclude using one or more predefined radius parameters to determine ifan image is within the geographic radius of another image based on thegeo-location codes on both images. Note that the geo-clusteringalgorithm may be required to account for the geo-location coding thatactually indicates the location of the camera instead of the location ofthe object or landmark. The geo-tagging of photographs may be achievedthrough several means including GPS-enabled digital cameras, GPS devicesseparate from the camera together with matching software, using a toolsuch as Google Earth, or manual editing of the photograph's ExchangeableImage Format (EXTF) tag. The methods of geo-tagging are generally knownin the art and are not described in this disclosure. Also, although adefault geographic cluster radius may be appropriate for most landmarksor objects of interest, some landmarks may require different clusterradius parameters in order to yield the most effective grouping ofimages. In stage 301, clusters of one or more images are generated basedon geographic proximity.

In the geo-cluster validation stage 302, each one of the geo-clustersgenerated in the create geo clustering stage 301 may be validated basedon selected criteria. For example, in one embodiment of the presentinvention, the goal may be to ensure that each geo-cluster selected forfurther processing reasonably includes a tourist landmark, i.e., apopular landmark. Accordingly, a validation criteria may be to furtherprocess only geo-clusters having images from more unique users than apredetermined threshold. A validation criteria such as having at least apredetermined number of unique users having submitted images of the samelandmark, is likely to filter out images of other buildings, structuresand monuments, parks, mountains, landscapes etc., that have littlepopular appeal. For example, an enthusiastic homeowner posting picturesof his newly built house of no popular appeal, is unlikely to post anumber of images of his house that is substantial when compared to thenumber of images of any popular landmark posted by all users of Internetdigital image collection sites. In one embodiment, the threshold may beset per season and/or per geographic area. In other embodiments, thethreshold may be derived by first analyzing the geo-clusters for thedistribution of unique users. In yet other embodiments, the thresholdmay be set for each type of landmark. The foregoing descriptions ofmeans for setting the threshold is only for illustration. A personskilled in the art will understand that there are many other meansthrough which the geo-clusters can be validated according to the focusof each use.

FIG. 4 illustrates further details 301 of processing in thegeo-clustering stage in an embodiment of the present invention. For eachgeo-tagged image, stages 401-405 may be repeated. For each geo-taggedimage that does not already belong to a cluster, the distance from theimage to each cluster is determined in stage 401. The distancedetermination may be based on the geo-coordinates of the center of theimage. For example, in one embodiment the distance may be from thecenter of the image to the moving average image center of a cluster,where the moving average is updated each time a new image is added tothe cluster and may be computed as the average of the centers of each ofthe images in the cluster. In stage 402, a decision is made as towhether the image matches an existing cluster. The decision may be basedon the geographic coordinates of the image falling within an areadefined by a predetermined radius from the center geographic coordinatesof the cluster. The predetermined radius may, for example, be based on aper geographic area basis, based on analysis of the center coordinatesof the images in each cluster, or be based on the type of landmark. Ifthe image is considered a match for a existing cluster, then it is addedto that cluster in stage 403. Otherwise, a new cluster is created instage 404. Adding an image to an existing cluster, or creating a newcluster, some cluster parameters may need to be calculated such as thegeo-graphic center coordinates for the cluster. When process 301completes for the input set of geo-tagged images, a set of geo-clustersshould be available. The geo-clusters, together with the associatedinformation, may be stored as part of the geo-tagged image corpus 110 oranother storage device accessible to the processing module 101. Theinformation associated with each image or geo-cluster may includegeo-location and other metadata describing images, text tags assigned toimages where available, and additional location information (i.e., textlabels specifying country and city) based on geo-location informationfor images.

FIG. 5 is a detailed view of the visual clustering stage 202 in anembodiment of the present invention. For each geo-cluster generated instage 201, stages 501-505 are repeated. The input to the visualclustering stage 202 is a set of geo-clusters produced in stage 201. Theoutput from the visual clustering stage 202, is one or more visualclusters for each of the input geo-clusters. Each visual cluster shouldinclude images having the same, for example, popular tourist landmark. Aset of visual clusters may collect all images depicting a particularlandmark in various camera angles, camera distances, and lightconditions. Whether this set of visual clusters contains all images andonly those images having a particular landmark is a function of theeffectiveness of the visual clustering method and parameters. Theteachings of this disclosure apply whether or not a set of visualclusters has all images and only those images containing a particularlandmark. For a geo-cluster, stage 501 creates an index of the images inthe cluster. The index may be a list of the images in the cluster,having data elements including the original image or a reference to theoriginal image, an image derived from the original image (for example,low resolution versions of the original image), one or more imagetemplates and feature vectors, user identification, geo-tagging, timeinformation, and any tags that have been assigned. In stage 502, eachimage in the geo-cluster is matched against the corresponding index. Thematching process 502 generates references to matching images, for eachimages in the geo-cluster. After the matching process 502, the index maycontain, for each image, references to all other matching images withinthat geo-cluster. The matching in stage 502, may include objectrecognition within each image to identify objects of interest such aslandmarks, generating feature vectors for each identified object, andthem comparing feature vectors to obtain match information. Thecomparison can be based on configurable numerical scores assigned tofeatures included in feature vectors, and configurable numericalthresholds to classify two images as a matching pair. Methods of objectrecognition in images and of generating feature vectors are well knownin the art. For example, methods of object recognition in images aredescribed in David G. Lowe, “Object recognition from localscale-invariant features,” International Conference on Computer Vision,Corfu, Greece (September 1999), pp. 1150-1157.

In stage 503, based on the index and the matches generated in stages501-502, a match-region graph is generated. In the match-region graph, anode is an image, and the links between nodes indicate relationshipsbetween images. For example, a pair of images that match according tostage 502 would have a link between them. The match-region graph isused, in stage 504, to generate the visual clusters. Briefly, a visualcluster is a connected sub-tree in the match-region graph, after theweak links are pruned based on additional processing in stage 504. Weaklinks may be, where images are matched based on image or featuretemplates, the links with less than a threshold number of matchingfeatures. Some embodiments may consider links that do not match aspecified set of features as weak links. Text label agreement, whereavailable, between images in a cluster may be another criteria. Also,the number of images in a cluster may be considered when pruning weaklinks so as to minimize clusters with very few images. A person skilledin the art will understand that pruning weak links may be based on avariety of criteria, in addition to those described here. Lastly, thevisual cluster data is saved in stage 505. The visual clusters may besaved to the landmark database 121. Along with the images and the objectinformation of each visual cluster, other pertinent data including butnot limited to, one or more text labels descriptive of the cluster, andone or more images particularly representative of the cluster, may besaved. A text label descriptive of the visual cluster may be generated,for example, by merging text labels of each constituent image of thatcluster. One or more images particularly representative of a visualcluster may be useful to display in an index, for example, of populartourist landmarks.

In another embodiment of the present invention, user verification of thegenerated visual clusters is implemented. FIG. 6 illustrates a graphicaluser interface 601 that may display the images in each visual cluster toa user, and provide the user the ability to manually edit variousaspects of each cluster. For example, graphical user interface mayretrieve visual clusters stored in the landmark database 621 and writeback the edited visual clusters to the same database 621. The graphicaluser interface 601 may include a cluster labeling module 602 that allowsa user to assign a new text label and/or modify currently assigned textlabels to each cluster and/or image. For example, cluster labelingmodule 602 may display each cluster with its current text label and thelabels assigned to individual images in the cluster, and allow the userto modify the text label assigned to the cluster. A cluster mergingmodule 603 may allow a user to merge or split clusters. Such manualmerging or splitting of clusters may be desired by a user after havingviewed the images in one or more clusters. A cluster editing module 604may allow a user to add or delete individual images from clusters.Module 604 may be useful in manually eliminating a poor representationof a cluster's corresponding landmark, as well as to manually add one ormore new images of a clusters corresponding landmark. In addition to theabove, embodiments of the present invention may offer the user variousoptions in interacting with the system 100.

Returning to FIG. 1, in some embodiments, a popularity module 104 maycompute a popularity score for each visual cluster, and rank the visualclusters accordingly. One or more of the indexes 122 used for accessinglandmark database 121 may be based on the popularity rankings computedby the popularity module. The popularity score of a cluster may be basedon, one or more of, the total number of images in the cluster, number ofunique users who have contributed images to the cluster, the number ofimages or images with unique user identifiers that are within a certainpredetermined radius of the center of the visual cluster. It should beunderstood that the popularity score may also be computed using othermethods not described above.

In another embodiment of the present invention, the landmark database isgrown incrementally. FIG. 7 is an exemplary process that may be used toincrementally grow the landmark database. Newly available geo-taggedimages are downloaded to local storage or made available to theprocessing module 101 by other means in stage 701. In stage 702geo-clustering is implemented over all available geo-tagged imagesincluding the new geo-tagged images. Geo-clustering was described abovewith respect to FIGS. 3-4. In stage 703, the geo-clusters resulting fromstage 702 are subjected to visual clustering. Visual clustering wasdescribed above with respect to FIG. 5. Having completed the visualclustering, in stage 704, some embodiments may propagate some or all ofthe changes initiated by the user on the previous clustering in thevisual clustering previously stored in the landmark database. Forexample, the user assigned or modified tags may be propagated to the newclustering. Optionally, in stage 705, the new visual clustering may besubjected to user verification and manual edit. Several types of userinteraction were described above with respect to FIG. 6.

The system 100, having a landmark database 121, may enable manyapplications. For example, the landmark database 121 may be used tosupplement user queries in order to make the queries more focused. FIG.8 illustrates a process that may be used to supplement user queries inone embodiment. A received user query may be parsed for a set ofpredetermined trigger words in stage 802. For example, city names suchas “Paris” may be used to trigger for landmarks in the city or viceversa. Having identified trigger words in the query, the landmarkdatabase may be searched in stage 803 for those trigger words toidentify associated tag words. Following the earlier example, a triggerword of “Paris” may cause the search to discover “Eiffel Tower”. Theassociated tag words that are identified are then used to supplement thequery string in stage 804. Such supplemented query strings may be usefulfor finding a broader spectrum of relevant information.

Another application, in one embodiment of the present invention, isshown in FIG. 9. Process 900 may be used for on-line automated taggingof digital images. For example, in stage 901 a new digital image iscompared to images in the landmark image database. If one or morematching images are found, then tags are generated in stage 902 based onall the matching images. In stage 903, the new image is tagged with thenewly generated tags.

FIG. 10 illustrates a user interface 1000 in an embodiment of thepresent invention where a set of landmarks is selected according to userinput, and details about the visual clusters of each selected landmarkare displayed. A landmark that is selected according to user-specifiedcriteria may be displayed within each area such as 1010. Each selectedlandmark may also have an area for receiving user input, for example,such as check box 1040. For each displayed landmark, a summary list ofthe visual clusters can be displayed. The summary list of visualclusters can be displayed such that it is clearly shown to belong to theparticular displayed landmark, for example, the summary list of visualclusters for the first displayed landmark can be contained within thedisplay area 1010 corresponding to the first displayed landmark. Eachentry 1020 of the summary list of visual clusters for a displayedlandmark can have a corresponding location to receive user inputspecific to that cluster, such as, for example, the checkbox 1030corresponding to the visual cluster represented in 1020. Each entry 1020can include descriptive information about the cluster 1022 and a link1021 to retrieve further details. For example, descriptive informationabout each cluster may include the number of images, popularity in termsof the number of unique users or authors contributing images to thecluster, information as to whether the cluster has been manuallymodified or verified, and any access information such as keys. The link1021 includes a linking method such as a user-navigable hyperlink toretrieve the images and individual image related data of the selectedcluster.

FIG. 11 is a flowchart showing the processing related to interface 1000in an embodiment of the present invention. In stage 1110, a userspecifies one or more selection criteria, such as, country, city,region, and/or other keyword. User-specified information, includingkeywords can be used to search for images based on tags assigned to theimages. The user may also specify other retrieval criteria such as aminimum level of popularity of the displayed landmarks, and landmarkshaving a minimum number of images submitted by users. For example, auser may want to view landmarks in Egypt for which at least 10 separateusers have submitted images. The user may also specify that onlylandmarks having at least a specified number of images should bedisplayed. Stages 1112 through 1120 are repeated for each landmarksatisfying the user-specified selection criteria. In stage 1112 one ormore landmarks satisfying the user specified selection criteria isfound. For each selected landmark, stages 1114 through 1116 are repeatedto display the visual clusters having the selected landmark. In stage1114 a visual cluster is selected, and in stage 1116 informationdescriptive 1020 of the visual cluster is displayed. For example, thenumber of images, the number of unique user identifiers or authors ofimages, a link to access the images in the cluster, other accessinformation etc., may be displayed for each visual cluster. For eachvisual cluster that is displayed in stage 1116, a user input graphic,such as, for example, a checkbox 1030 can be displayed and enabled foruser input.

In stage 1118, a determination is made as to whether there are morevisual clusters to be displayed corresponding to the selected landmark.If no more visual clusters are to be displayed for the selectedlandmark, then in stage 1120, information about the landmark isdisplayed. For example, information such as the name and location of thelandmark, popularity, number of images etc., can be displayed. For eachlandmark displayed in stage 1120, a corresponding user input graphic mayalso be displayed and enabled for user input. For example, in FIG. 10, acheckbox 1040 may receive user input corresponding to the landmarkdisplayed in area 1010. In stage 1122, a determination is made as towhether there are additional landmarks to be displayed. If all landmarksthat satisfy the user specified selection criteria have been displayed,then in stage 1124, user input corresponding to visual clusters isreceived. The user input corresponding to visual clusters may indicate,for example, that one or more clusters are to be merged, or that one ormore clusters are to be disassociated from the selected landmark. Instage 1126 the visual clusters are processed accordingly. In stage 1128,user input corresponding to each landmark is received. The user inputcorresponding to each landmark may indicate, for example, that one ormore landmarks are to be merged and/or deleted.

FIG. 12 shows a user interface 1200 in an embodiment of the presentinvention where a user can view information about a selected visualcluster. The interface 1200 may include an area 1210 where one or moreexample images representative of the selected visual cluster aredisplayed, an area 1220 in which a group of descriptive data elementsincluding details of each image in the visual cluster are listed, and anarea 1230 in which a selected image is displayed. The area 1220 mayinclude descriptive information 1224 and corresponding user inputgraphic, such as check box 1222, for each image in the selected cluster.The descriptive information 1224 may include, for example and withoutlimitation, a link to retrieve the corresponding image, data and timeinformation for the image, author information for the image, and taginformation. The area 1230 can display an image retrieved from the listdisplayed in 1220. The image displayed in area 1230 may enable the user,for example and without limitation, to view the region of interest 1232in the displayed image. The ability to ascertain the region-of-interestin any image, for example, may allow the user to better determine thesuitability of the particular image being in the current cluster.

FIG. 13 is a flowchart showing the processing related to interface 1200in one embodiment. In stage 1310 user input is received selecting avisual cluster. In stage 1312, one or more images representative of theselected visual cluster is selected and displayed, for example, in area1210. In stage 1314, information for each image in the selected clusteris displayed, for example, in area 1220. The information listed for eachvarious data elements including, for example and without limitation, alink to retrieve the corresponding image, data and time information forthe image, author information for the image, and tag information. A userinput graphic, such as, for example, a checkbox 1222 may also bedisplayed for each listed image and enabled for user input. In stage1316 user input is received. In stage 1318, the visual cluster isprocessed according to the received user input. For example, images canbe deleted from the selected cluster, some tag information can bechanged, etc.

In an embodiment of the present invention, the system and components ofthe present invention described herein are implemented using well knowncomputers. Such a computer can be any commercially available and wellknown computer capable of performing the functions described herein,such as computers available from International Business Machines, Apple,Silicon Graphics Inc., Sun, HP, Dell, Compaq, Digital, Cray, etc.

Any apparatus or manufacture comprising a computer usable or readablemedium having control logic (software) stored therein is referred toherein as a computer program product or program storage device. Thisincludes, but is not limited to, a computer, a main memory, a hard disk,or a removable storage unit. Such computer program products, havingcontrol logic stored therein that, when executed by one or more dataprocessing devices, cause such data processing devices to operate asdescribed herein, represent embodiments of the invention.

It is to be appreciated that the Detailed Description section, and notthe Summary and Abstract sections, is intended to be used to interpretthe claims. The Summary and Abstract sections may set forth one or morebut not all exemplary embodiments of the present invention ascontemplated by the inventor(s), and thus, are not intended to limit thepresent invention and the appended claims in any way.

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

1-5. (canceled)
 6. A method comprising: receiving, by a computer, a userquery; identifying, by the computer, one or more trigger words in theuser query; selecting, by the computer, one or more corresponding tagsfrom a landmark database corresponding to the one or more trigger words;supplementing, by the computer, the user query with the one or morecorresponding tags, generating a supplemented user query; retrieving, bythe computer, one or more landmarks based on the supplemented userquery; generating, by the computer, a user interface including the oneor more retrieved landmarks; and causing one or more summary lists ofvisual clusters for the one or more retrieved landmarks to be displayedon the user interface, wherein each summary list corresponds to one ofthe one or more retrieved landmarks and includes descriptive informationand access information for each cluster.